2.2 Sufficiency

1 Definition

Sufficiency is a central concept in that it allows us to focus on the essential aspects of dataset while ignoring irrelevant details.

Statistic, Sufficiency

  • A statistic T(X) is any function of data X (not including parameter).
  • A statistic T(X) is sufficient (for model P) if the conditional distribution of X|T(X) is the same for all P∈P, i.e. independent of θ.

In short, sufficient statistics carry all information about θ.

2 Factorization Theorem

Theorem (Factorization Theorem)

Let P={Pθ|θ∈Θ} be a model with densities pθ(x) with common measure μ. Then T(X) is sufficient iff ∃gθ(t),h(x)≥0, with pθ(x)=gθ(T(x))⋅h(x) for almost every x under μ.

Another example is orde statistics. For X1,⋯,Xn∼i.i.dPθ, and any model P={Pθn|θ∈Θ} on X⊂R. if Pθn is invariant to permutation of X=(X1,⋯,Xn) (see exchangeability), then S(X)=(X(1),⋯,X(n)) is sufficient.

3 Minimal Sufficiency

For the example of N(θ,1), we showed that ∑i=1nXi is sufficient. Then 1n∑i=1nXi is also sufficient.
Some sufficient statistics represent more significant compressions of data than others. Like X― can be recovered from S(X) but not other way around.

Proposition

T(X) is sufficient. T(X)=f(S(X)). Then S(X) is sufficient.

Minimal Sufficient

T(X) is minimal sufficient if

  1. T(X) is sufficient.
  2. T(X)=f(S(X)) for any other sufficient S(X). (almost surely in P)

We say x,y∈X are equivalent (denote as x≡py) if pθ(x)pθ(y) does not depend on θ.

T(x)=T(y)⇒x≡py.
Theorem

T(X) is minimal sufficient if x≡py⟺T(x)=T(y).

Q.E.D.

3.1 Minimal Form

Minimal Form

Form of pη(x)=eηTT(x)−A(η)h(x) is minimal if ∀η∈Ξ, T(X) satisfies no linear constraints, i.e. there is no nonzero vector a∈Rs and b∈R, s.t. ηTa=b,∀η∈Ξ,or T(X)Ta=P−a.s.b.

Otherwise we can represent P as an r− dim exponential form for some r<s.

Proposition

If pη is a minimal form, then T(X) is minimal sufficient.

The converse of this proposition is not true.

3.2 Diagram

For case s=2, let's consider the following example:
Pasted image 20241206202340.png|400